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We explore the potential for discovery of an exotic color sextet scalar in same-sign top quark pair 
production in early running at the LHC. We present the first phenomenological analysis at colliders 
of color sextet scalars with full top quark spin correlations included. We demonstrate that one can 
measure the scalar mass, the top quark polarization, and confirm the scalar resonance with 1 fb~^ of 
integrated luminosity. The top quark polarization can distinguish gauge triplet and singlet scalars. 
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Introduction — The Largo Hadron Collider (LHC) at 
CERN opens a new energy frontier in searches for new 
physics beyond the standard model (SM) of particle 
physics. According to the current LHC operation sched- 
ule, a data sample based on ~ 1 fb~^ of integrated lu- 
minosity at 7 TeV center of mass energy is anticipated 
within two years. Signs of new physics (NP) would be 
observed at this luminosity if (i) the collider signature is 
novel and easily detectable; (ii) the cross section is rela- 
tively large; and (iii) the SM backgrounds are suppressed. 

In this Letter we present an analysis of a color sextet 
scalar model which satisfies all three key points, and we 
discuss the requirements for discovery and verification of 
tlie model. The final state of special interest to us is 
a same-sign top quark pair. We sliow that the mass of 
tlie new colored scalar can be estimated well and that 
its scalar nature can be verified by a measurement of the 
spin correlations of the final leptons from the decay of the 
top quarks. Among other consequences, observation of 
color sextet scalars would alter the evolution of the strong 
coupling strength and indicate a fundamentally different 
picture for unification from conventional grand unified 
theories. We exploit the valence quark content of the 
initial protons at the LHC by focusing on a production 
mechanism in which the quark-quark interactions play an 
important role in the production of a heavy resonance for 
which large values of the partonic Bjorken-a; are needed, a 
region in which the gluon parton density drops off rapidly 
but the quark density in a proton remains large. 

A bosonic colored state can be produced in quark- 
quark fusion with color structure obtained from 3x3 = 
6 © 3, where 3, 6, and 3 are the triplet, sextet, and an- 
titriplet representations of the quantum chromodynamics 
S'C/(3)c color group. Color sextet scalars are present in 
partial unification [l| . The masses of the scalars can be 
possibly as low as the weak scale, ~ TeV or less [3|. The 
fermion number violating interaction of a new scalar $ 
with two quarks is [3| 
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where i^^j, is the Clebsch Gordan coefficient Q of the 6 or 
3 representation, a and h are the color indices of quarks, 
Pl/r is the usual left- or right-handed projector and. 



without loss of generality, the Yukawa coupling \^/^ is 
real in this study. The coupling A^/^ can be relatively 
large since it is not proportional to the quark mass. The 
coupling of the new scalar to two up-type quarks is con- 
strained by the measurement of D'^ — D^ mixing and by 
the rate of 13 — >■ 7r+7r°(7r "'"(/)) decay, resulting in stringent 
bounds on the coupling A^^': |Re(AccA*„)| ~ 5.76 x 10^'' 
for 771$ ^ 1 TeV Q- However, this strong constraint 
can be relaxed if the coupling to the second generation 
quarks is minimized. We concentrate on the flavor con- 
serving couplings and treat A„„ and Xu as independent 
couplings. Alternatively, the first generation may be sup- 
pressed while the second generation is not, but such sea- 
quark initiated processes are relatively suppressed. 

A color scalar can carry the following electrowcak 
quantum numbers: (6,1,4/3), (6,1,-2/3), (6,1,1/3), and 
(6,3,1/3), where the numbers in the parentheses denote 
(5C/(3)c,5C/(2)L,[/(l)y); see Ref. [1,1,0 for details. 
The SM gauge symmetry demands that <&6,i,4/3 interacts 



with urur, $6,1,-2/3 with dndR 
urcLr, and $6,3,1/3 with [y/2' 



$6,1,1/3 with ulcIl and 

iULUL,-ULdL,V^dLdL) 11, 

where {uL,dL) denotes the SU{2)l quark doublet while 
ur and dR are the corresponding right-handed singlet 
fields. A measurement of the spin of quarks from $ de- 
cay would decipher the handedness of the coupling of 
the scalar to quarks and also the gauge quantum number 
of $. With this goal in mind, we consider the decay of 
$ — >■ ii in this work because a top quark decays promptly 
via the weak interaction such that its spin information is 
retained among its decay products. We emphasize that 
top quark polarization can be used to distinguish gauge 
triplet from gauge singlet scalars because triplet scalars 
decay to left-handed top quarks while singlet scalars de- 
cay to right-handed top quarks. We concentrate here on 
the specific case of $6,1,4/3 which couples only to right- 
handed gauge singlet up- type quarks, but our work can 
be easily extended to include a study of $6.3,1/3- 

The most striking collider signature of a color sex- 
tet scalar is same-sign top quark pair production in a 
direct s-channel process, as is pointed out in Refs. [7[ 
where top quark decay is not considered. It is important 
to retain the top quark spin correlations in the calcu- 
lation because the kinematic distributions of the decay 
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FIG. 1: Leading order cross sections (pb) via uu — >■ $ and 
uu — >• $* at the Tevatron (dotted) and at the 7 TeV LHC 
(sohd and dashed) for \uu = 1- We use the CTEQ6L parton 
distribution functions [3| and choose the renormalization and 
factorization scales as m^,. 



products depend strongly on the top quark spin. The 
new scalar interacts only with right-handed gauge sin- 
glet quarks so the top quarks from its decay are mainly 
right-handed. A charged lepton from a right-handed top 
quark decay has a harder momentum spectrum than one 
from a left-handed top quark. No available event gener- 
ator calculates color sextet scalar induced tt production 
and subsequent t decay with full top quark spin correla- 
tions included. We calculate the helicity amplitudes for 
uu -^ tt -^ 6M^+(^ l+v)bW'^{-^ £+!/) and implement 
the process in our own parton-levcl Monte Carlo code. 

Because the decay width of (f> is narrow, r($ — > qq) w 
Yg— 7n$A^„, one can factor the process uu — > tt into scalar 
production and decay terms, 
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The decay branching ratio Br(tt) = Br($ -^ tt) is 
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Subscript "0" denotes Xuu = A^ = 1. We choose to work 
with the following two parameters in the rest of this pa- 
per: the scalar mass m^ and the product A^„Br($ — > tt). 
The kinematics of the final state particles are determined 
by the scalar mass, whereas the couplings of the scalar 
to the light and heavy fermions change the overall nor- 
malization. 

Figure [T] displays the $ production cross sections via 
Mw ^ $ and uu -> $* at the Tevatron (dotted) collider 
and at the 7 TeV LHC. The uu initial state dominates 
uu for a pp initial state. Therefore, tt pairs are produced 
much more than tt pairs, leading to an asymmetry of the 
same-sign charged leptons. This doubly positive excess 
would be an early hint of a color sextet scalar. 

The search for the same-sign top quark pair produc- 
tion in the dilcpton mode at the Tevatron imposes an 



upper limit a{tt + tt) < 0.7 pb Q. The CDF collabo- 
ration measured the tt invariant mass spectrum in the 
scmilcptonic decay mode [lO|. Since b and b jets from 
t -^ Wb are not distinguished well, tt pairs lead to the 
same signature as tt in the semileptonic mode. Hence, 
the rriff spectrum provides an upper limit on a{tt + tt), 
shown in the cyan shaded region in Fig. [21 The lower gray 
shaded region is the region in which $ would hadronize 
before decay, washing out the spin correlation effects we 
utilize to probe the coupling and spin of the sextet state. 
Event simulation — We focus on the same-sign dilepton 
decay mode, in which the W bosons from both t — >■ Wb 
decays lead to a final state containing an electron or 
muon, W -^ Iv, accounting for about 5% of all tt decays. 
Wc concentrate on the clean fJ.'^p,'^ final state because 
the muon reconstruction has a large average efficiency 
of 95 — 99% within the pseudorapidity range |?7| < 2.4 
and transverse momentum range 5 GeV < pt ^ 1 TeV, 
while the charge misassigned fraction for muons with 
PT = 100 GeV is less than 0.1% ^. 

These events are characterized by two high-energy 
same-sign leptons, two jets from the hadronization of the 
&-quarks, and large missing energy (Pt) from two un- 
observed neutrinos. The dominant backgrounds yielding 
the same collider signature are the processes (generated 
here with ALPGEN [13): 

pp -^ W+{^ e+iy)W+{-^ e+iy)jj, (4) 

pp -^ ti ^ bW+{-^ £+iy)b{^ e+)W-{-^ jj). (5) 

The first process (WWjj) is the SM irreducible back- 
ground while the second (tt) is a reducible background 
as it contributes when some tagged particles escape de- 
tection, carrying small pt or falling out of the detector 
rapidity coverage. For example, one of the 6-quarks de- 
cays into an isolated charged lepton while one of the two 
jets from the W~ boson decay is mistagged as a 6-jet. 
Other SM backgrounds, e.g. triple gauge boson produc- 
tion {WWW, ZWW, and WZg{-^ bb)), occur at a neg- 
ligible rate after kinematic cuts, and are not shown here. 
At the analysis level, all signal and background events 
are required to pass the acceptance cuts pt.j,pt,i > 
50 GeV, \7jj\ < 2.5, \r]i\ < 2.0, and Ai?jjjf,« > 0.4, where 
the separation in the azimuthal angle ((/))-pseudorapidity 
(77) plane between the objects k and I is AR^i = 



iVk —fli) + {i'k — (t>i) ■ We model detector resolution 
effects by sm earing th e final state energy according to 
SE/E = A/./eJGcV®B, where we take A = 10(50)% 
and B = 0.7(3)% for leptons(jets). To account for b- 
jet tagging efficiencies, we demand two 6-tagged jets, 
each with a tagging efficiency of 60%. Wc also ap- 
ply a mistagging rate for charm-quarks Cc-s-b = 10% 
for pt{c) > 50 GeV. The mistag rate for a light jet 
is eu,d,s,g^b = 0.67% for pt{j) < 100 GeV and 2% for 
PtU) > 250 GeV. For 100 GeV < pr (j) < 250 GeV, we 
linearly interpolate the fake rates given above |13l |. 

After lepton and jet reconstruction, we demand two 
hard leptons of the same sign, a requirement which 



TABLE I: Signal and background cross sections (pb) before and after cuts, with Xuu ~ Mt = 1, for six values of m$ (GeV). 
The decay branching ratios of the signal Br(ff) are given in the second column. The "no cut" rates correspond to all lepton 
and quark decay modes of W-bosons, whereas those "with cut" are obtained after all cuts, the restriction to 2 ^"'"'s and with 
tagging efhciencies included. 
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FIG. 2: Discovery potential of the process iiii —^ $ —> tt at 
the 7 TeV LHC with £ = 1 fb"\ The shaded regions are 
excluded by the Tevatron measurements: the orange band is 
derived from the search of same-sign top quark pair produc- 
tion; the cyan band denotes the 95% C.L. constraints from the 
rriti measurements. The gray region indicates that $ would 
hadronize before decaying, washing out spin-correlations. 



greatly reduces the SM background, giving a rejection 
of order lO"'^ and 10~^ for the tt and WWjj processes, 
respectively. After the cuts are imposed, we find a total 
of 4.6 background events, 3.2 from ti and 1.4 from WWjj 
for 1 fb~^ of integrated luminosity. After 6-tagging and 
restriction to the /i^/i^ mode, more than half the signal 
events survive the analysis cuts. Signal and background 
cross sections arc shown in TableU before and after cuts, 
for 6 values of r?T,$ . 

In Fig. [21 we show the expected numbers of signal 
events as a function of m^ for a range of values of the cou- 
pling A^„Br((f> -^ tt). We obtain the event rate lines by 
converting the required cross section into A^„Br($ — ^ tt) 
via Eq. [2] In a similar fashion, the 5cr discovery line 
is found by requiring 14 signal events. Rates for other 
values of Xuu and Xu can be obtained from Eq. [2j 
Reconstruction and Measurements Given a sam- 
ple of events having two like-sign muons, two 6-tagged 
jets, and missing energy, we now address the possibil- 
ity of establishing that these events are consistent with 
$ ^ tt ^ bW+{-^ i+iy)bW+{-^ e.+v). A full event re- 
construction is required. We demonstrate that the Mt2 
variable can be used for full kinematic reconstruction and 
to identify the W^-bosons and top quarks in the interme- 
diate states. 



The Mt2 event variable is designed to bound the 
masses of a pair of heavy particles that subsequently de- 
cay into one or more visible states and missing energy. 
It is a function of the momenta of two visible particles 
and the missing transverse momentum in an event [14 1. 
Strictly speaking, MT2(a, &7^t) is the minimum of a 
function 



max { Mt {pt,M, Mt [pt J2)] 



(6) 



such that ^1 7^+ p2T =/Et- Here a and h are the two 
individual (or clustered) visible states from the parent 
decay, and ^j^, jjj a-re the associated missing momenta. 
The transverse mass Mt is defined as 
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where X denotes the visible particle or cluster. 

The Mt2 approach requires the correct assignment of 
final state particles to the a and b clusters if the parent 
particle is to be reconstructed successfully. For the W- 
boson, the assignment is straightforward (one ^^ — >■ a 
and the other fi^ -^ b), and it yields an endpoint at the 
W^-boson mass, confirming the two /i"*" leptons and ^t 
originate from the decay of Ty-bosons. However, there 
are two possible combinations of /i-5 clusters for recon- 
structing the t-quark mass. To choose the correct com- 
bination, we require the fi-b pairing to be such that Mt2 
is minimized. This choice picks the correct combination 
with 95% probability. The same method is used success- 
fully in Ref. |15| to determine the top quark mass from 
a ti sample, with both t quarks decaying leptonically. 

Once the correct pairing of b and ^"'" is made, the en- 
tire event can be reconstructed. There are 6 unknowns 
from ^^ and ^2^ ^'^o of which are eliminated by the ,Et 
measurement. The 4 unknowns may be resolved via the 
four on-shcU conditions of W bosons and top quarks, re- 
sulting in a four- fold solution [ifll . Two of these solutions 
are real, providing momentum determination of the two 
neutrinos. The top quarks from the $ decay may there- 
fore be fully reconstructed, allowing the mass of $ to be 
determined via the tt invariant mass, as shown in Fig. [3^. 

Given the top quark four-momenta, the spin of $ could 
in principle be found via the polar angle distribution of 
the top quark in the reconstructed rest frame of <!'. How- 
ever, distinguishing the spin-0 case from a spin-1 can be 
difficult with limited statistics. Instead, one may probe 
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FIG. 3: (a) Normalized tt invariant mass distribution for 
m,s> = 600 GeV and Xuu ~ \tt ~ 0.3 after the neutrino 
momenta are reconstructed, (b) Distribution of the angle 
of the charged lepton relative to the top quark in the cm. 
frame before (red dotted) and after (black) cuts and efficien- 
cies are included. Separation cuts distort the distribution only 
slightly, allowing one to read the top quark polarization with 
few events. Distributions shown for 1 fb^^ of simulated data 
for the signal only; the SM background is negligible. 



the coupling structure and spin of the $ via the polar- 
ization of the top quarks. Among the decay products of 
a top quark, the charged lepton is well correlated with 
the top quark spin direction |17| . In the helicity basis, 
the polarization of the top quark can be found from the 
distribution in ^hei, the angle of the lepton in the rest 
frame of top quark relative to the top quark direction 
of motion in the overall cm. frame. The angular corre- 
lation of the lepton £+ is given by ^(1 ± cosflhoi), with 



the (4-) choice for right-handed and (— ) for left-handed 
top-quarks. In Fig. [5}d, we present this normalized dis- 
tribution for a right-handed top quark before cuts (red) 
and after cuts (black) with 1 fb~^ of simulated data for 
a scalar mass of to$ = 600 GeV. The cuts produce some 
distortion, but the linear dependence on cos ^hci is robust 
and certainly allows discrimination from i(l — cos^hci)- 
For instance, the cos 6^+ distribution after cuts peaks in 
the direction of the reconstructed top quark, cos^hei ~ 1, 
since the hard lepton pT from the top quark decay more 
easily passes the lepton threshold cuts. Based on Pois- 
son statistics, the distribution in Fig. [5]d can be distin- 
guished from an unpolarized flat distribution with as few 
as 0(30) events at 95% C.L. spread over 3 bins, making 
this measurement possible in an early LHC scenario. 

If both top quarks exhibit the same polarization in the 
rest frame of <&, either i^ii or i/ji/j, one can conclude the 
resonant $ state is a scalar rather than a vector, as a color 
sextet vector would decay into 1^1 jj Q. The SU{2)l 
quantum number of $ is also uniquely determined: a 
triplet for t^tL while a singlet for tfftfj. 
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